A. The Field of the Invention
The present invention is directed to systems for evaluating the performance of large scale programmable machines capable of having a plurality of terminals connected thereto. More particularly, the present invention is directed to a method and a system, referred to in the art as a robot, which emulates the actions of human operators and is used for evaluating the performance of large scale multi-user host computer systems.
B. The Background Art
As modern society has come to rely more and more upon the use of programmable machines such as digital computers for business, governmental, scientific, and other applications, the variety of available hardware and software has increased dramatically. For example, only a few years ago an organization looking to acquire a high performance computer system would have had to choose from only a few vendors. As use of digital computers has proliferated throughout society, the number of vendors providing hardware has also rapidly expanded.
With the number of possible choices always on the increase, purchasers of large scale computer systems have come to feel an urgent need to determine whether a particular configuration of computer system hardware will perform as specified. Very often, such large scale computer systems are used in organizations where a plurality of terminals, which may include devices such as video terminals, keyboards, bar code readers, and other devices, provide communication with the CPU of the computer system. In such situations, measuring the period of time between when a user, such as a human operator using a bar code reader, inputs information to the CPU and the time required by the CPU to provide an appropriate response is extremely important.
Not only does the response time effect the number of transactions which can be handled over a specified period, and thus the profitability or efficiency of the organization, but long delays between inputting required information and receiving appropriate responses causes dissatisfaction and aggravation to the human users of the system at each terminal.
Since the acquisition of a large scale computer system is a major capital item in any organization, the area of computer performance evaluation has received increasing attention in recent years. By utilizing computer performance evaluation techniques before purchase of a large scale computer system, it is hoped that the large scale computer system which is chosen can provide the best performance at the lowest possible cost.
Moreover, computer performance evaluation techniques are also used by both hardware and software vendors to satisfy contractual and customer requirements that the computer system performs at its minimum specifications before the customer takes delivery. Thus, there has been an effort in the art to provide computer performance evaluation systems and techniques to test the performance of hardware, software, and combinations thereof.
Vendors of computer hardware often times provide potential customers with specifications indicating that a particular configuration of some large scale computer system will carry out "X million instructions per second (MIPS)" or some other criteria of computing speed and power. Unfortunately, such specifications provide little reliable information concerning the performance of a large scale computer system having many, perhaps hundreds or thousands of communication ports, each having a terminal connected thereto and also requiring an interface with peripheral devices such as magnetic disc drives and other devices.
The economic incentives of being able to accurately predict whether a combination of computer hardware and software will perform as specified has lead to the development of various approaches and techniques to carrying out computer performance evaluations.
The performance of computers is a function of many interrelated considerations, including the programming tools available, the size of the problem to be solved, the algorithm used to solve the problem, the computer operating system's efficiency, and the architecture of the computer. Generally, three approaches are used to conduct a computer performance evaluation (CPE). These three approaches include: (1) simulations; (2) benchmark testing; and (3) monitoring.
Simulations are a technique of performance evaluation whereby a large scale computer system to be tested is simulated using software programs. When utilizing simulations, it is necessary that both hardware and software functions be simulated by the simulation program. Thus, a program code must be written to simulate functions such as magnetic disc input/output, terminal input/output, data base handling, operating system's functions, and application program functions.
Unfortunately, the use of simulation techniques requires that the simulation accurately quantify the performance of each of the components in the large scale computer system being simulated. Without such accurate quantifications, attempts to predict the large scale computer system performance under actual operating conditions becomes impossible. Thus, disadvantageously, a great deal of effort must be spent by programmers to create a simulation program for each computer system to be evaluated. It will be appreciated that if a potential customer desired to estimate the likely performance of ten different large scale computer systems available from different vendors, the amount expended in creating ten different simulation programs (one for each potential computer system) may far outweigh the benefits which come from carrying out the performance evaluation. Moreover, the inherent inaccuracies and uncertainties involved in simulations is ever present.
Another computer performance evaluation technique is referred to as benchmark testing. Benchmark testing utilizes a group of computer programs (sometimes referred to as a suite) to test specific performance capabilities of the host computer system. In benchmark testing, it is very important to define what exactly is being tested by each of the computer programs. Significantly, more often than not, benchmark test programs evaluate the quality of the programming code in which they are written rather than the performance of the host computer on which they run.
In a multi-user environment, three main factors that effect transaction throughput are: (1) multi-programming level, (2) data sharing among simultaneous transactions, and (3) the transaction mix. In a benchmark evaluation, it is possible to develop statistics on multi-user capability of a host computer system by running a series of data base inquiries in a single user mode and then again in a multi-user mode. The results of such benchmark testing can then be extrapolated to estimate the performance of the complete system with all terminals in place and operational. For example, benchmark programs may evaluate the difference in performance of the host computer system when the number of users increases from one to ten. The difference in performance may then be extrapolated to estimate the performance of the host computer system when 100 users are placed on a system.
Unfortunately, such extrapolation seldom provides an accurate picture of host computer system performance under actual operating conditions. For example, benchmark testing results when using only a few users cannot be extrapolated to predict the performance of the host computer system when a full load of 400, 600, or more, user terminals are added to the system. This is due to the fact that many computer systems perform well up to a particular work load with their performance dropping rapidly as that particular work load is exceeded. Thus, benchmark computer performance evaluation techniques, while more desirable than simulation testing, have major drawbacks and disadvantages when an accurate prediction of host computer system performance under actual operating conditions is expected.
The inadequacy of other methods of computer performance evaluation has lead to the development of various techniques for monitoring a large scale host computer system under actual operating conditions. For example, a user determining which computer system to acquire, or the configuration of computer system after a particular vendor has been selected, would, in one form of monitoring testing, actually assemble the complete host computer system with all peripheral terminal devices attached thereto, for example 600 user terminals, and load the CPU with actual application programs and data and have all 600 terminals being used as if under actual operating conditions.
Some forms of computer performance evaluation monitoring testing require hardware and/or software which has been specifically developed for a particular computer system in order to measure the transaction throughput which occurs under actual operating conditions. Disadvantageously, the costs of such hardware and/or software for monitoring transactions through the computer system under actual operating conditions is prohibitive for users evaluating computer systems before purchasing the same.
Moreover, it will be appreciated that organizing a monitoring test under actual operating conditions is a major logistical feat when a large number of terminals are involved. While the loading of data and application programs into a host computer system is a relatively simple procedure (since the applications programs and data will change relatively little from machine to machine), the connecting of 600 terminals to the CPU and coordinating the 600 human operators for the duration of the evaluation session is an immense task. Moreover, since human operators are used to input information, the input rate at which information is given to the host computer system cannot be accurately controlled.
It will be appreciated that monitoring tests of a large scale host computer system having a large number of terminals connected thereto (e.g., 100-1000 terminals) under actual operating conditions is a very time consuming and expensive task. While such monitoring tests produce evaluations which justify a high degree of confidence therein, the costs and difficulty of performing such monitoring tests make the alternative one which is seldom carried out.
As will be appreciated from the foregoing, it would be considered an advance in the art to provide a system for efficiently evaluating the performance of a large scale multi-user computer system. It would also be an advance in the art to provide a system for carrying out computer system evaluation monitoring more efficiently than previously possible. It would be a further advance in the art to provide computer performance efficiency monitoring system which may be particularly adapted for use with a changing number of communication ports which are active on the host computer system. It would be a still further advance in the art to provide a computer performance evaluation system which is readily transported from one location to another and set up for use.
Still another advance in the art would be to provide a system for computer performance evaluation monitoring under full load conditions without requiring human operators or terminal devices being associated with the host computer. It would be a further advance in the art to provide a system for computer performance evaluation monitoring which allows monitoring tests to be accurately repeated. It would be still another advance in the art to provide a system for computer performance evaluation monitoring that can be used with a large number of different host computer systems with little modification.
It would be yet another advance in the art to provide a system for carrying out computer performance evaluation monitoring tests that is cost efficient to use on host computers from a variety of vendors. It would also be an advance in the art to provide a system for computer performance evaluation monitoring which can vary the timing at which transactions are presented to the host computer system as well accurately vary the transaction rate. It would be still another advance in the art to provide a computer performance evaluation monitoring system which records the time taken to complete each transaction.